On 23/05/15 10:45, Don Pearce wrote:
On Fri, 22 May 2015 16:55:20 +0100, Jim Lesurf
wrote:

In article
wwv1ti8r5fm.fsf@l1AntVDjLrnP7Td3DQJ8ynzIq3lJMueXf 87AxnpFoA.invalid,
Richard Kettlewell wrote:
These paths don't identify a particular kind of fileystem; they identify
what kind of files should be put there. In particular, /tmp (or
$TMPDIR) is a location for temporary file.

If the operator wants, or does not want, to use a tmpfs for temporary
files, that's up to them. There's no such thing as a path which is
guaranteed to be a tmpfs; what filesystems are used where is down to
local policy.

That sums up why I decided I couldn't use this method for programs I
release for others to use. Its a shame, though, because it potentially
means the programs are slower and may 'wear' the SSD/HD.

It seems a shame, though, given that RO makes this easy.

Jim

Can you wear an SSD this way? As I understand it they have an internal
OS that is constantly shifting the data around anyway, and putting in
some more from outside really doesn't make a great deal of difference.

d


I would not worry about SSD wear - I have been using them in laptops for
a while - my current one is over a year old and shows 100% on the
Wear_Leveling_Count (that's good, 0% is "dead").

Jim Lesurf writes:
Richard Kettlewell wrote:

These paths don't identify a particular kind of fileystem; they
identify what kind of files should be put there. In particular, /tmp
(or $TMPDIR) is a location for temporary file.

If the operator wants, or does not want, to use a tmpfs for temporary
files, that's up to them. There's no such thing as a path which is
guaranteed to be a tmpfs; what filesystems are used where is down to
local policy.

That sums up why I decided I couldn't use this method for programs I
release for others to use. Its a shame, though, because it potentially
means the programs are slower and may 'wear' the SSD/HD.

I don’t understand your objection. As far as I can see you’re choosing
to avoid even the possibility of a tmpfs, on the grounds that you might
not actually get it. Which seems like cutting off your users’ noses to
spite their faces.

--
http://www.greenend.org.uk/rjk/

In article , Martin Gregorie
wrote:
On Fri, 22 May 2015 14:57:59 +0100, Jim Lesurf wrote:


AIUI tmpfile() will give me a file 'handle' (FILE*) but no name. How
would I use that as part of the command string for flac?

I did find a man page for tmpname() but having given the details it
says to "Never use it!".

There are a couple of other ways to do this:

1) use a script to run flac and pipe its output directly into your
program like this:

flac inputfile | yourprogram


Your program will use a loop to copy bytes from flac's stdout stream
into a temporary file created using something like:

#include stdio.h ... FILE flacfile = tmpfile();

This is a nameless file which will vanish when you close it or exit
from the program. You'd fill it by reading stdin and after that you
can reposition to its start by calling rewind() and read through the
file as many times as you want.


That looks like worth trying. However I'm not clear on details so would
like to check some points:

1) When calling flac using system() can I just replace the flac command's
final switch/setting

-o outputfile

with something like

inputfile | RunImage

(The program by default is being run with a process name 'RunImage',
although I can of course change that. Or can I get and use a process
number?)

Will the output be readable as a stream of bytes/chars I can use to fill a
buffer?

Will it pipe out until system() is done, so will be readable afterwards?

And is it the case that simply using

FILE flacfile = tmpfile();

will mean it will read as stdin when I use flacfile as the 'handle'?

Can I also use fread() of the correctly sized block rather than read as a
series of bytes/chars/etc?

Is there an example I can see of this? I'll do a websearch, but as yet
don't know anything about it!

Thanks again, this is interesting. I've know of the existence and general
pupose of pipes, but no idea how to use them.

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

On Sat, 23 May 2015 09:45:45 +0000, Don Pearce wrote:

Can you wear an SSD this way? As I understand it they have an internal
OS that is constantly shifting the data around anyway, and putting in
some more from outside really doesn't make a great deal of difference.

'Constantly' is a bit rich - a block is only moved when flash memory
writes require it to happen. So the short answer is that every write to
flash storage reduces its remaining life.


--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |

In article , Tim Watts
wrote:
Can you wear an SSD this way? As I understand it they have an internal
OS that is constantly shifting the data around anyway, and putting in
some more from outside really doesn't make a great deal of difference.

d


I would not worry about SSD wear - I have been using them in laptops for
a while - my current one is over a year old and shows 100% on the
Wear_Leveling_Count (that's good, 0% is "dead").

I'm not particularly worried about SSD wear as I've also (so far) used some
for years. But some of the people who use my program may be using
conventional spinning rust. And there *are* wear mechanisms for SSD
according to reports I've read. Just that in practice I (and many others)
don't seem to have run into problems with it.

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

In article
wwvpp5rpql7.fsf@l1AntVDjLrnP7Td3DQJ8ynzIq3lJMueXf 87AxnpFoA.invalid,
Richard Kettlewell wrote:
If the operator wants, or does not want, to use a tmpfs for temporary
files, that's up to them. There's no such thing as a path which is
guaranteed to be a tmpfs; what filesystems are used where is down to
local policy.

That sums up why I decided I couldn't use this method for programs I
release for others to use. Its a shame, though, because it potentially
means the programs are slower and may 'wear' the SSD/HD.

I don't understand your objection. As far as I can see you're choosing
to avoid even the possibility of a tmpfs, on the grounds that you might
not actually get it. Which seems like cutting off your users' noses to
spite their faces.

"Objection" perhaps isn't the correct word. My concern is about how to let
the user employ it given that the 'locations' they have for tempfs may
vary.

FWIW I'm now experimenting with having my program use a tempfile whose
name is given via a user setting.

The default is to use the interal-to-application directory, as before. But
by giving an alternative that location and filename can be used.

So I've been trying

/run/user/1000/complicatedname

where the is a name that seems to be unlike the exising examples and
fairly randomised and long. But which also refers to the program name in
part. I can see there is a risk that the name may clash with that wanted by
something else. But I've tried to make the risk low.

This speeds up the program operation quite noticably. Particularly useful
for 'high res' files that can have hundreds of megs of LPCM to churn
though. (Note I'm doing this in 5-sec chunks, so the total temp storage is
more limited, but the end-to-end amount can be fairly high.)

However I did this because /run/user/ followed by my user id number seemed
a reasonably directory choice given that 'df' shows me its a tempfs
location mount *on my system*.

This mimics fairly well the approach I'm used to on RO.

*But* AIUI I can't assume that every user of every distro will have *that*
place 'free and available' as a ram-based temp storage they can user-acess.

So my "Objection" is in essence not being able to rely on a chosen
'location' name like /run/user. For RO I could in effect simply use 'ram:'
followed by a program-specific tempname (again settable by the user if
necessary).

Hence my current approach of allowing a user to pick a place if they wish
to have the tempfiles in ram. Or to leave as default if they're happy with
that.

That's about the simplest change my limited understanding makes possible at
this stage. It seems reasonable to me given that any user wouldn't need to
do anything if the default works and they are happy. If they want a faster
operation and dodge any 'wear' they can decide having used df and poked
about, where to choose.

But I will look at pipes.

All that said, if anyone else wants to do an improved version for their own
use or release, I'd be delighted. My interest is in the *results* I and
others can get. The program is a means to that end.

Once I've tested it a bit more and added relevant info to the Help I'll
make the tweaked version available. Then let any users decide as they
prefer.

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

Jim Lesurf writes:
Richard Kettlewell wrote:
If the operator wants, or does not want, to use a tmpfs for temporary
files, that's up to them. There's no such thing as a path which is
guaranteed to be a tmpfs; what filesystems are used where is down to
local policy.

That sums up why I decided I couldn't use this method for programs I
release for others to use. Its a shame, though, because it potentially
means the programs are slower and may 'wear' the SSD/HD.

I don't understand your objection. As far as I can see you're choosing
to avoid even the possibility of a tmpfs, on the grounds that you might
not actually get it. Which seems like cutting off your users' noses to
spite their faces.

"Objection" perhaps isn't the correct word. My concern is about how to let
the user employ it given that the 'locations' they have for tempfs may
vary.

You’re looking at this completely wrong. People don’t have locations
for tmpfs. There’s no such concept. People have locations for
temporary files, and their own policies about how temporary files are
stored, based on their preferred tradeoffs between performance,
persistence, etc.

However I did this because /run/user/ followed by my user id number seemed
a reasonably directory choice given that 'df' shows me its a tempfs
location mount *on my system*.

It’s a terrible choice. /run is for pidfiles, sockets, etc. It happens
to be a tmpfs in some configurations but there’s no guarantee that this
will be true.

It’s not that long since /run didn’t even exist.

--
http://www.greenend.org.uk/rjk/

In article
wwvbnhbpixk.fsf@l1AntVDjLrnP7Td3DQJ8ynzIq3lJMueXf 87AxnpFoA.invalid,
Richard Kettlewell wrote:
"Objection" perhaps isn't the correct word. My concern is about how to
let the user employ it given that the 'locations' they have for tempfs
may vary.

You're looking at this completely wrong. People don't have locations
for tmpfs. There's no such concept.

Perhaps you missed my use of inverted commas. :-)

Would it help if I'd said locations in the filing system tree that are in
user accessible ram and use tempfs? If so, I'm happy to correct my wording.
Even if that wording is wrong, I hope that by now what I mean should be
clear.

People have locations for temporary files, and their own policies about
how temporary files are stored, based on their preferred tradeoffs
between performance, persistence, etc.

OK, please cf below on that.

However I did this because /run/user/ followed by my user id number
seemed a reasonably directory choice given that 'df' shows me its a
tempfs location mount *on my system*.

It's a terrible choice. /run is for pidfiles, sockets, etc. It happens
to be a tmpfs in some configurations but there's no guarantee that this
will be true.

Erm, that was actually one of the points I was making as a reason that made
this problematic for me. And hence why any such choice would be a user
option they'd have to make *if* it suited them. They'd have to decide what
alternative to /run.... suited them for they system.

It's not that long since /run didn't even exist.

The fact that it is 'new' doesn't seem like a reason for never using it.
However...

So what would you suggest in terms of a filing system address like
/run/user.... ? If your answer is "none of them" then I will stick with
what suits me and leave others to do what they prefer. But if there is a
common and recommended 'location', I can use it.

I must admit that I find it curious that despite the very logical
"everything is a file" approach of *nix that it seems to lack a standard
filing system address like /ram which gives the individual user some of
their own user space on ram. Its a standard feature of RO which is a minor
OS in comparison to Linux. It seems strange no-one has picked up on the
idea as being useful for users of Linux who aren't highly experienced
programmers. Or have they, but I don't know the address to use? Being so
used to the simple convenience of RO 'ram:' (and memphis) it seems an odd
situation.

However all that said. No-one is compelled to use any of the programs I've
written. No-one who uses this example is compelled to take up the option.
No one who does is forced to use /run/user/1000/whatever as the tempfile
'directory'. etc. And anyone who wishes is very welcome to do what they'd
feel is a "better" version if they want. At all levels they can make their
own choices.

As I've said, I'll try to learn about pipes, etc. And I may change when I
learn more. But the reality here at present is that what I've done seems to
work nicely for my purposes. And I thought one of the strengths of Linux is
that we can each use the system as we prefer.

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

Jim Lesurf writes:
So what would you suggest in terms of a filing system address like
/run/user.... ? If your answer is "none of them" then I will stick with
what suits me and leave others to do what they prefer. But if there is a
common and recommended 'location', I can use it.

As I said, use $TMPDIR if set, otherwise use /tmp. It’s a standard
location for temporary files and is often a tmpfs.

If you want your own /tmp to be a tmpfs, do that.

If you want a guarantee that your program will always have a tmpfs
available at some predictable location: sorry, you can’t have one.

As I've said, I'll try to learn about pipes, etc. And I may change
when I learn more. But the reality here at present is that what I've
done seems to work nicely for my purposes. And I thought one of the
strengths of Linux is that we can each use the system as we prefer.

You can do what you like, as I believe I’ve already said. But if you
want your program to integrate well with other people’s systems, there
are right and wrong ways to go about it.

--
http://www.greenend.org.uk/rjk/

In article
wwvzj4vnuuk.fsf@l1AntVDjLrnP7Td3DQJ8ynzIq3lJMueXf 87AxnpFoA.invalid,
Richard Kettlewell wrote:
Jim Lesurf writes:
So what would you suggest in terms of a filing system address like
/run/user.... ? If your answer is "none of them" then I will stick
with what suits me and leave others to do what they prefer. But if
there is a common and recommended 'location', I can use it.

As I said, use $TMPDIR if set, otherwise use /tmp. It's a standard
location for temporary files and is often a tmpfs.

I'll need to unpack that because of my ignorance.

My mint laptop (for example) has /tmp. But 'df' doesn't list it, so does
that mean its on the main SDD simply as a directory. i.e. *not* a tempfs
(or ramfs) on ram?

If you want your own /tmp to be a tmpfs, do that.

That I don't currently know how to do. Nor do I know if it might give a
problem for something else. There are already two directories inside /tmp.


If you want a guarantee that your program will always have a tmpfs
available at some predictable location: sorry, you can't have one.

I take it that by "can't" what you really mean is that "the user would have
to be willing to create one. Which in turn means I'd have to explain /
request that *if* I wanted them to use a tempfs"?

However I think I pointed out that I'm not saying they must have this. All
I've done with the new version of the program is allowed the user to
specify where the tempfiles will go. And by default this will be as before
- inside the actual application's directory. Personally, I will go on to
prefer a ram based tempfs, but others can easily do otherwise.

As I've said, I'll try to learn about pipes, etc. And I may change
when I learn more. But the reality here at present is that what I've
done seems to work nicely for my purposes. And I thought one of the
strengths of Linux is that we can each use the system as we prefer.

You can do what you like, as I believe I've already said. But if you
want your program to integrate well with other people's systems, there
are right and wrong ways to go about it.

It seems reasonable to me to give them the choice to do what they prefer.
That's what my new version does.

However for myself (and as a suggestion to others) it seems to me to make
sense to use tempfs for this program's purpose.

Neither of the above seem "wrong" to me. But others can choose what they
want.

To go on...

I've found that I can do something like

sudo mkdir -p /mnt/tmp

followed by

sudo -t tmpfs -o size=20m tmpfs /mnt/tmp

to create a temporary 20meg 'ram disc' mounted at /mnt/tmp

Is that a more sensible option for me (and others who want to get the speed
and avoid disc wear) to employ?

I've noticed that the /mnt/tmp then persists being listed over a power
cycle, but the mounting/creation of the 20meg tmpfs space seems to be
needed after each bootup. So on from that, I assume I'd need to add
something appropriate to fstab if I want to avoid having to do that every
time?

Again, does that not seem reasonable?

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html