On Wed, 20 Jan 2021 11:29:47 +0000, Chris Elvidge wrote:

> On 20/01/2021 10:49 am, Martin Gregorie wrote:
>> On Wed, 20 Jan 2021 10:03:49 +0000, Pancho wrote:
>>
>>> On 19/01/2021 22:15, Martin Gregorie wrote:
>>>
>>>>> I think you are missing the point. If I pipe 4095 characters into
>>>>> mawk,
>>>>> nothing happens, if a pipe an extra char to make 4096, it prints
>>>>> out.
>>>>
>>>> Thats definitely faulty behaviour: pipe operation should not depend
>>>> on how full the pipe is.
>>>>
>>>>
>>> Richard explained it better than me. It's mawk waiting until it has a
>>> block of 4096 bytes (or EOF). Clearly designed behaviour.
>>>
>>> With 20.04, Ubuntu seems to have switched from mawk to gawk as default
>>> awk. I don't know if this is Ubuntu specific or Debian. So it's quite
>>> possible this will be reflected in the next version of Raspbian
>>> (Rasberry Pi OS)
>>>
>>> This is exactly what I mean by fragile. Someone writes a script to do
>>> something using awk, an OS update comes along and the app completely
>>> changes.
>>
>> In Fedora systems the binary is called gawk with awk as an alias In
>> Raspbian Buster the binary is /usr/bin/mawk with awk and nawk as
>> aliases
>>
>> So, the same shell script should run in both places: my test scripts do
>> exactly that *and* do not show the long delay you're seeing.
>>
>>
>>
> In mine, Linux raspi-3plus 5.4.79-v7+ #1373 SMP Mon Nov 23 13:22:33 GMT
> 2020 armv7l GNU/Linux,
>
> # ls -l /usr/bin/?awk -rwxr-xr-x 1 root root 537K Sep 14  2018
> /usr/bin/gawk*
> -rwxr-xr-x 1 root root  93K Apr  8  2012 /usr/bin/mawk*
> lrwxrwxrwx 1 root root   22 May 27  2020 /usr/bin/nawk ->
> /etc/alternatives/nawk*
>
> so mawk is well out of date!
>
> /usr/bin/awk -> /etc/alternatives/awk -> /usr/bin/gawk
>
> /usr/bin/mawk -W version says:
> mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan
>
> compiled limits:
> max NF             32767 sprintf buffer      1020
>
> So really, well worth linking awk to gawk if you can.

I think that requires the Debian maintainers to switch from mawk to gawk
and then everybody else to wait while the new codes percolates up through
the distro dependencies.

However, I have realised that the *content* of the input file may be
affecting awk, since its designed to work with lines of text.

I started out by grabbing a suitably large JPG image and adjusting its
length using the 'truncate' utility, which makes files of exactly the
specified size. Then I read its manpage a little more carefully and
discovered that, if it is used to lengthen a file, the new space is
filled with binary zeros: something awk may not handle well since its
expecting textual input.

As a result, I wrote another awk script for creating files of pseudo-text
of exactly the specified length. Each line in a generated file is up to
37 characters:

0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ\n

and can also generate very short final line if the required amount of
data isn't an exact multiple of 37:

empty   ''  (empty)
1 char  '\n'  (1 char
2 chars '0\n'
3 chars '01\n'
...

The test results I posted were all the result of running a command like
this:

"cat filename.txt | awk -- 'script'"

that uses 'cat' to fill a pipeline with one of the files created with my
awk file filling program and 'script' is an awk program that totals the
lengths of the lines read and displays the total.

So, questions for Pancho:

- how did you create your input files?

- Is there any chance that they don't contain any 'newline' characters?

I'm asking because its quite possible that awk would stall, waiting for a
the next character, if it had read several KB of data without finding a
newline character or EOF.


--
--
Martin    | martin at
Gregorie  | gregorie dot org

--- SoupGate-Win32 v1.05