Organized and Chaired by Luke Kanies
Notes by Jim Thornton (corrections to
<jthornton@parc.com>)
LISA
'05 Site; Notes from previous years:
2003
2004
Tuesday
Evening, December 6, 2004
Town and Country Resort, San Diego, CA USA
These are notes from a freewheeling BoF
discussion, taken in real time. Treat them as paraphrases not quotes and note
that attributions are particularly sketchy.
Luke: My name is Luke Kanies, been doing this for a while ... Run/am server automation company, currently trying to sell it to companies etc.
Last year one of the big questions was "What is configuration mgmt?" Hasn't got a lot clearer: Server automation but centralized but big picture or something like that. Want to brainstorm on some other ways to talk about this. At the configuration mgmt workshop Tom Limoncelli had a talk about how badly we market, argues configuration management is a bad term: too long, too many syllables.
Anything to start?
Jeff McCune: We're running into cfengine [problems?] are there any object oriented tools?
Narayan Desai: you'll find that both Luke and I have written tools.
Jeff: Can one do classes in cfengine?
Luke: Modelling not that different, talk about same things, get much
more complicated in cfengine. My tool (puppet) influenced greatly by
shortcomings I saw in cfengine, also influenced by bcfg (Narayan's tool)
?: Would you say that bcfg is OO?
Narayan: We have an aspect system in middle ... ends up nicely composable system. Things end up being fairly terse, works fairly well, admins seem to be happy.
Kent Skaar: I work for a commercial company called BladeLogic, we have things as objects but can't really do subclassing.
Paul Anderson: Lots of systems support various degrees of OO. Differences between systems, take SmartFrog does the OO very well but might not be what you would want to use as end-user tool [?]
Mark Verber: Using inheritance isn't necessarily the most intuitive model, especially in a world trying to use multiple inheritance. For semi-naive person without full model of inheritance rules they can end up hanging themselves if e.g. not good discipline about where overrides and method substitutions are used.
[...]
Paul: Problem with all these OO frameworks: really forcing an inheritance hierarchy on your data which is very hard to find a good inheritance hierarchy. Really need aspect composition model: good one has multiple hierarchies of things. Systems do that in varying ways at the moment.
Narayan: Hit important point: compositional semantics is biggest weakness that is readily apparent.
?: Change subject, more practical questions: I've got cfengine (insert favorite tool name there) where if server member of plone server group then becomes member of zope group, for use becomes member of http server group, then maybe sql server. ... Really would like data to be in LDAP or MySQL DB or something like that and gets pulled in: figure out channel X gets ...
Luke: A lot of people have done this: basically use the addclasses function: command that looks up with whatever mechanism you want.
?: Need list of server names?
?: No, just need to know what classes of servers you expect to have. Using exec or whatever you can make external call to verify that I am this class of server, then can use all syntax etc.
Luke: If you go to cfengine list people have this. cfengine wiki probably has something.
Luke: How many using cfengine? Those who did not say cfengine, what are you using?
Paul: Use tool called lcfg (lcfg.org) Configuration tool we've
developed running as research testbed and production tool for us for about 10
yrs. Still think it is a pretty good tool and has a lot of
good things in it but has never had the effort put in to turn into a product but
is not the sort of thing you would take on lightly though it has a lot of good
concepts in it.
Luke: One thing we don't talk about a lot, but it came up in
workshop
... we think of Configuration Management (CM) as how you tell the computers what
they should do. I think monitoring and other aspects of managing ... if you
could
configure correctly you wouldn't need to monitor.
Mark Verber: [interrupts] Not true, there are anomalous conditions, ... I'm in a world where I know what all the bits are on machines, never touched by humans but still monitoring. Also need auditing function.
Luke: Do you consider this part of CM?
Mark Verber: We have a central store ... old company TellMe networks
Verdad system, first version was open source hopefully enhancements out soon.
That holds description of config and has series of things that use that ... e.g.
service routing all based out of same config store. Monitoring, logging,
and projects ongoing to (ultimately about delivering a service and not
configuring hosts) very important to be
able to take things in and out of service: non-stop environment need to be able
to do rolling updates. Now reaching into load balancers etc. One thing
uncompleted: way to specify data flows between various
service components. Ideally want that to lock down hosts and networking gear.
[...]
Mark Verber: Verdad 2 project in theory inside TellMe trying to address a lot of limitations. Originally done very flexibly since we didn't know what we were shooting for. Also packaging system Platinum and Guild. At core was something we called "Appliance Model of Computing". Search for Brad Porter (O'Reilly emerging tech talk and Stanford EE380 advanced topics pot pourri last spring) [ed. http://www.stanford.edu/class/ee380/Abstracts/040407.html]
Luke: Taking same code at new company? That was great, going to do that again?
Mark Verber: New company is very small, one thing we're doing is Knowledge Representation, looking to dogfood our story ... will see in a few months whether it is a good idea or not.
Luke: Any other homebrew systems out there that you love?
?: Mention Radmind
Luke: config.sage.org is a
simple mailing list I set up last year: very little traffic because (did I say)
nobody knows how to define it. There's a CM community but not a practicing
community where people
swap ideas about how to solve their problems. Forum is there but conversation is
not. ... likely to move to LOPSA. Also FAQ that I maintain there, one of
few regularly updated things about this. Every tool listed here except Verdad is
listed there, plus simplistic definitions, everything around the topic, even
tools not traditionally considered like rsync. ... Companies are convinced that
everything we do is proprietary and can't be shared.
Paul: Do you want to put up lssconf URL? ... google for lssconf [ed. http://homepages.informatics.ed.ac.uk/group/lssconf/]
Luke: lssconf is the list associated with the workshop: pretty successful as a theory list. I created the other list for practical discussion.
Paul: Web page has all notes from workshops.
Jeff: So for homegrown stuff, from my perspective current stuff handles managing services and leaves managing resources up to human admin. My question: we home wrote some scripts that more manage resources. One thing we do is move workstations around all the time. Most of the time everything is keyed off of hostname: that was prohibitive because the admin had to manage tie between hostname and machine. We home-grew provisioning system to enroll machine in DB and then link to hostname. Are there any open source or commercial products that do that for you? ... Bootstrap by putting machine on private net, netboot; as part of enrollment it puts most hardware info in a DB, then I can link that [physical] machine to a [logical] host. I can link a pool of resources to a pool of services.
Luke: Does DB include CPU speed etc.?
Jeff: Not yet, but now, e.g. Mac gets whole config. That leads to second
question, should monitoring be part of site config or not? Theoretically the
monitoring could look at used resources to run services and look at tie between
service pool and resource pool.
[...]
Luke: I would say that you should publish that. There's very little work on how to take data you have encoded in one place to another place ... In the same vein I've got a simple thing called 'factor': I can teach it to find info, e.g. hostname, and more complicated things like MAC address. If you've got 50 ways to find a domain on 70 platforms you can teach it all these ways. That can be used to encode the kind of info you want to collect. If you've got a better library I would be happy to replace mine with yours. We all have these tools that we think are small and so we don't publish. I really try to focus in my development on segregating functions as much as possible into libraries, daemons whatever so you can publish something. ... At least somebody else can reuse.
Narayan: There's a need for systems like this in cluster systems (provisioning). That's someplace where there is probably an audience for the thing you have written.
Luke: I would like config.sage.org to turn into a place to talk about the tools you are writing. People in CM started hacking scripts ...
Bartosz Ilkowski (University of Buffalo, Center of Excellence in Bioinformatics): In my personal opinion the approach can be different. Tools out there are written for different purposes but do the same thing, e.g. resource management is critical. ... There's a variety of commercial tools, open source tools. Historically most written for enclosed environments which may not be readily usable in open environment with far more servers exposed to external environment. I would imagine that taking time to fix those would be less time spent. That would give you immediate access to a whole view of the infrastructure which you are trying to manage. ... Can do baselining from that and track deltas etc. Just an example. ... Mechanisms are out there used for data provisioning: it doesn't matter if you provision for bioinformatics or for installing packages for apache server.
[discussion about issues in building clusters]
Narayan: One node is going to be node 1. ... Some amount of work about
those mappings. The thing you have written sounds really interesting, interested
in how it fits in ... things aren't actually
static.
Bartosz: Nonetheless those things could be used in a wider array of systems.
Kent: You described something very traditional about high level admins: you glue together whatever you tools you find and it is crude and an enormous integration effort. Even using it in your own environment.
Bartosz: If someone has an alternative it is better to learn it than to do your own.
Luke: "A couple of months of development will often save us a couple
of hours on the net" [derived from a saying from laboratory science]. ... I
think it would be beneficial if we spent more
time downloading tools even if you throw it away. Send some feedback to the
person, even better put some feedback on the net. That would be really helpful.
Andrew Cowie (Management Consultant): Two things come to mind:
Luke: You can do something. Every tool on this list I have downloaded and tried to write a config and start a daemon ...
Narayan: I want to build on what Andrew said: in my experience System Administrators (SAs) are judgmental ... when you try to sell something that changes every part of how you work if you download it and it segfaults once then you're done.
Luke: cfengine belies that (bad code, core dumps) but it is the major tool.
Mark Verber: How many places are using it?
Luke: I encounter people all the time who are using it but not talking about it.
Andrew Cowie: But it has a community around it. Communities grow around things. I'm speculating that one of the reasons you've been pushing a rope is there is nothing to coalesce around.
Luke: Talking about Puppet community: very successful getting people conversing (15 people on IRC channel) and generically active, people doing other projects. I did release early, talk about warts, have a blog.
Andrew Cowie: I believe in "release early, release often" but it takes a certain kind of guts.
Jeff: With puppet, you may have an advantage where it is targeting an existing user base that has a tool that works for them.
Luke: No. I have a client where I hired everyone in the Unix group ... and they still won't use it.
Jeff: Seems like a resource manager, people are very careful.
Luke: [gives example of RDtool]
[...]
Kent: RDtool was not a tool that did anything to your system.
Jeff: How many people have a test network that they test on before they deploy live?
Luke counts about 4.
Jeff: How do we do this?
Luke: Tools need to help you: if I were to say "go" what would you do? Do some introspection to figure out the delta between what I want to be there and what is there now. As you write your tools you should understand that introspection, debugging etc. are very important.
Paul: Can you explain that? Are you trying to test whether the tool is broken or whether it is not good enough to let you say what you want?
Luke: Often you write up what the network should look like and you are wrong. ... New tool in existing environment.
Narayan: Need an impedance match there. The other thing is you need to
convince admins that they should trust this tool. After they see enough traces
then they will believe that it is ok. I've been working on bcfg for 3 years and
getting very close to the point where it is documented well enough to use. Still
dealing with little details. The only way we'll know if it works is if people
use it because it works for us.
That's the problem isn't it? You have this tarball that works great until
you hand it out to somebody and it does nothing. The thing you were saying
about SmartFrog is good example: we know from documentary evidence that it is a
really powerful tool that does a lot that other things can't do. There was a
movie! But you spent three hours trying to build it and gave up.
Luke: Twice
Ed Smith: When did you try it?
Luke: About this time last year and around this time the year before.
Ed Smith: They did have very good install instructions.
Luke: Might have been installing it on a platform that didn't use Java in the way they expected.
Narayan: The paper I'm presenting tomorrow is about the process of taking a tool into a group and environment that already exists and what it takes to get admins to trust it and integrate it into work on a daily basis. It was hard.
Luke: Non-technical right?
Narayan: Lot of issues were rooted in technical issues, can talk tomorrow in CM session.
Ed Brown (LANL): We've been using cfengine 4-5 yrs (before I started working there). Still only couple of people who understand it well, up to 20 we're trying to get to use it. Can you talk about introducing a tool, getting them to use it, understand CM tools. It's beyond a lot of people.
Luke: One of my previous clients has fully automated CM and never touches machines manually except when they go behind the two people who do touch machines.
Guy Ferraiolo (CNET): I'm not an SA but performance guy. I run small performance lab and so have a pretty serious configuration problem... What I find is that the more competent the SA the more they resist because a lot of times they are craftpersons and don't want to automate that away.
Narayan: I disagree slightly. I agree that the most competent ones are craftspersons. ... We've been viewing it as taking repetitive tasks out so admins make decisions they are best qualified to make.
Guy: I agree but was talking about mythology.
Narayan: We need to start viewing CM tools differently. Efficiency is a big part of it but the appeal of that to SAs is negligible except at the extreme end of the scale (e.g. 2000 machines with 1 admin). ... People are afraid of being idle. You don't end up with half the admins [if you do this] but the users are happier because sometimes they see you.
Guy: I'm more concerned with correctness, 1 machine in 16 is different.
Mark Verber: When I've worked on CM that was a key feature: an older place had systems that should have behaved the same but "one of these things is not like the other". One of the easy sells where I am now, is that the way the developers build, the way QA works, all have the same tools, all use the same engines. That's the only way we can be sure that what the engineers are building actually ends up in the production world.
Luke: What makes it harder for all level of sysadmins: most tools have
only one input mechanism and one output mechanism. One big config, you need to
understand the whole thing. Not terribly modular.
... My personal belief not tested yet is that if you had more ways to interact
with the system (e.g. high-level web based interaction to find out what is going
on, then increasing detail and precision) that would help people transition. You
don't have to use it, just look at it. You have trending etc. When management
cares it changes things. Nobody likes to be ignorant because the CM system is
new to me. Need to build a gentle ramp.
Mark Verber: Are the tools sufficiently complex that they are obscuring what people are really shooting for?
Luke: They don't match networks well.
?: Are there ways to make them easier?
Luke: Yes, the focus of Puppet. e.g. keys in cfengine, initial install needs to be easy. As a developer you need to focus on it.
Narayan: Going back to what you're talking about, you have 1-2 people who understand whole thing ... Get to the person who doesn't know anything but has problem to solve right now and their job is to solve the problem not to use the tool. We need tools that deal with this. We need to figure out how to actually provide some sort of smooth transition from what you've got to what you want. Our approach with bcfg: impedance match that tells you the difference between what I've got and what I want. Everybody needs to be able to make changes.
?: When I've been trying to promote tools developed in-house.
?: In terms of adoption we're discussing the wrong thing again. Most
of these tools aren't so complicated that you couldn't write them. The
problem is that people don't think systematically. You should be
preaching systematic management. There are tools but you don't have to use them.
... They say "this tool will force me to do all these things" where the tool
should be an enabling thing. Sure we could make
tools better, sure we should do that. The point is that people aren't ready to
manage things in a systematic way.
?: Disagreement: people will agree in theory.
Kent: No, Narayan has the opportunity to be in an environment where they have already decided to automate out of necessity. Ed is describing what we have seen when people buy a product suite: they have already decided to do something systematic with the environment. The biggest win sometimes is just to go down a systematic approach.
Luke: A lot of shops aren't interested in systematic and I don't sell to them. There are a lot of shops who do but don't have tools etc. The vast majority of shops are not doing systematic management but nobody is providing them a very good example.
Narayan: That's why I wrote the paper.
Luke: What I would love to see on the mailing list is notes about what people have tried, not material at a peer-reviewed paper level.
Matt [?] (Student): I think I disagree with you two on the point of being prepared to accept systematic processes. It's probably true to some extent but doesn't account for so many people writing in-house stuff instead of using existing tools. Even though cfengine has a community, people go through writing their own.
Paul: Two questions: why write your own and why use something else? In many cases, there are no good tools at the moment. People write their own tools because that is the SA tradition. I think we won't move forward in CM in that way. The kinds of things you need to do are too big for sysadmins to hack up as amateurs. We haven't got a professional community doing that, this is an in-between state.
John Warwick (University ?): Even if you've got a commercial tool that was supposed to solve your problem you then spend 20 hrs configuring it. People write a tool because they think they can spend the 20 hrs on their own and then they understand what they have.
Narayan: It's easy to get into it: I though it wouldn't be hard and that was 3 years ago.
John: I think you need a quick entry for a site.
Paul: That has nothing to do with CM.
Luke: Maybe not your definition but it is something we need to do.
Paul: The job of system configuration is to put that software on those machines to behave like those guys want. That doesn't say anything about config files or anything.
Andrew Cowie: That is IT: CM is a subset of what you just said.
Narayan: Where the rubber meets the road: Paul is describing where we need to end up. What you're describing is you want a tool to fix something. The problem is that to get from where you are to where he wants to be, the model of how you're going to solve the problem is completely different. Changing how you do things is non-starter [?]. ... How people perceive problems changes with tools.
Luke: [argues that this is a lot about abstraction levels]
?: If there is a question between writing scripts vs. adopting other stuff to do what you need, the better admin (or well managed) will search for something to get done what you need. The problem I have is that we keep trying to describe a new paradigm and we don't get to the point of being able to describe it even to the more experienced. I keeping coming to these things and fail to grasp the paradigm in a way that can be communicated. That's the dead horse we keep beating. The same problem rears its head often.
Narayan: I agree but I think we need to do incremental steps.
Luke: He's not talking about tools.
[...]
?: Say I'm at a place and take [incremental] steps and my staff turns over and nobody saw the steps?
Narayan: The analogy I like is in the 60's when people wrote machine code. Programmers were extremely skilled .... higher level tools make things harder, harder to find people initially but once you build up a culture and code base, so to speak, you have more portability because now a lot of niggling details are under control that you don't need to write.
Jim [?]: We've [the world] done this a few times: e.g. moving from specific to general. I suspect that they [those who moved us to high-level languages] could define the problem pretty well with writing everything in machine code and could point to a solution. I'm not hearing that on CM.
Paul: No, it took a long time to evolve. It took theory and academic work and didn't come from the machine code programmers.
?: The problem is that nobody can pick up a tool and see how it works.
Jim: I'm talking about not something you can hold in your hand but a new way of doing things. ... [thinks programming management knew what they wanted]
Paul: [disagrees]
Narayan: We've learned over 30 years why you don't hire guys to write assembly. The usability model is really subtle, the only way to figure it out is to get a lot of users. The thing most valuable about the process was that I had a dozen people going through the process and when it didn't work they would come and pound on my door. I don't know until you get real people involved other than you trying it out.
Mark Verber: Just because a CM system is managing this doesn't mean it is harder for the user. At TellMe, we had 3 generations [of tools], thinking about the 4th. Especially at the beginning the people managing the system found their job became much easier partly because the abstraction model matched what they were doing.
Narayan: Doing something different doesn't mean harder but does make it harder to hire people.
?: Most of these tools have one developers.
Andrew Hume: But with open source we use the masses out there.
[breakdown into multiple conversations around this controversial though likely facetious statement]
Narayan: The thing about open source is that whenever we hear from
vendors that "we're open sourcing our code and we expect the community to pick
it up" [they are dropping it]
... Works sometimes but not a good default.
Luke: Ed's basic point was: no single person is so good that they can solve this problem by themselves. ... open source vs. commercial doesn't matter so much.
Mark Verber: There's certainly a size you need to get to for energy and ideas but that isn't a large group. Look at the research community e.g. Berkeley: Patterson would say 2-3 faculty, 8-12 grad students, a couple of staff members.
Paul: The problem is the mix of skills you need: understand admin, software etc.
Mark Verber: You shouldn't expect this to be solved by a large group, that will fail. You need a small group with right skills.
Andrew Hume: One of problems is you have to consider is what the payback is. If an academic does it there needs to be a PhD or serious papers or an NSF grant coming out. The bottom line is the reason you did the work [speaking to Paul] is that the problems had to be solved. So the energy is not there to generalize.
?: "Not Invented Here" (NIH) is a problem: everyone wants to go out and build a better mousetrap. We're trying to manage all these different mousetraps that are slightly different.
Luke: I tried every mechanism: open source, commercial, waiting for somebody to do it. A lot of this is out of desperation. I don't think there's a single config tool out there (with possible exception of [?]) that has more than one group with commit access. You need to have a couple of people with commit access.
[rumbles of disagreement about NIH]
?: Different twist: not so much that the tool is NIH but I've already got a big pile of stuff.
?: Agree, already committed to certain point and don't want to go back to 0
Mark V[?] (American Greeting): I like the idea of multiple commit access but most projects need some momentum and traction before they go with multiple commit access.
Luke: I think there's something special about the complexity of this area.
Ken Bier (Deutsche Bank) Some problems but at the end of the day you need a tool that runs on all kinds of stuff. We tried to go the vendor route (BigFix) and we helped them extend from Windows to AIX etc. Then suddenly they said 64 bit is never going to work etc. I don't think a vendor is going to step in unless they have a way of opening source code.
Luke: This is why I'm doing the company, I think it needs commercial support. ... I'm going to hire somebody who has to work on it.
Andrew Hume: One of the things that is quite different about this
space than nearly every other sort of tool is that the amount of code you have
to write to adapt to a local environment is quite large. It is
plausible to think of a generic system but there's an awful lot of particulars.
A large amount of work to individuate it to your site.
Kent: [interrupts] We haven't figured out what we can have in common across environments. Talking about what is different is silly until we talk about everything that is the same.
?: Naming standards, differences over /usr/local vs. /opt [cataloging classes of differences]
Andrew Hume: The amount of work you need to do to adapt is significant
or seems that way even if it isn't. ... If that bunch of work is commensurate to
what you've already done or what you think you can
hack up then there is really little motivation to put on your hair shirt and do
the work. That's the perception and you've got to do something to somehow
deflect, diminish, make non-true this situation.
Luke: For most tools (except cfengine) there are two parts :
I want a CPAN for system administration
Andrew Hume: One problem: to solve is the problem that you actually have to think about what you are doing. To quote somebody famous: "There is no labour to which a man will not go to avoid the labour of thinking". ... So many arbitrary puny little decisions each of no importance but each has to be specified.
Luke: Like somebody at dinner who just wanted to eat and the waitress had many questions: "no you can't just have food you have to answer all these questions first."
?: Like Andrew says, you get that first server tossed at you and now you have that as history. Second, third, it grows, it's hard to step back to say "what's the right thing to do?"
Andrew Hume: That's why Paul says that of all the properties you specified for apache I really only care about two: I trust you to specify them [the rest].
Luke: [another plea for people to join the mailing list and send something]
Jeff: Not quite 3 words, but we're talking about static tools vs. a framework. We need clear way for tools to talk to each other so we can pick and choose what we need but they all interoperate.
?: Look at all the different config files, all these different formats
from apache to sendmail to postfix. cfengine is interesting to me because it
looks like it will let me slowly pull it together but it is
hard. Sometimes you have to beat people over the head to do it. It is
always easy to do the quick and easy thing.
Andrew Cowie: What do people experience trying to beat people into submission? E.g. don't give admins root on production machine and require use of the tools.
Andrew Hume: I once was doing this thing remotely where I had access for a month and then after a month I was supposed to have no access. It was a good exercise to focus on not what you need to do but on how to recover when things screw up.
Luke: [reference to a talk about hacking silicon] One of the points
was that it is so ridiculously expensive to make these chips that everybody
builds in debugging because you can't afford to be wrong. ... Think about our
tools and how little introspection our tools provide and how little debugging
there is, full debugging harnesses. There's a much higher quality you demand
from your tools in situations like that
when you're going to go hands off. We're used to holding the hands of our
tools all the time.
Mark Verber: A lot of people believe that the 80% solution is the sweet spot but don't realize that 20% has an 80% leg. Development cost is fixed cost and the other is ongoing cost that never ends. Other thing is that part of the issue is not focusing on CM but looking at how software components behave. The observation that Andrew made is that in Apache there are 50 variables that have to be set and he only cares about 1 or 2. Maybe we're fundamentally building components in the wrong way and need to reduce variance.
Andrew Hume: One thing that made 1127 [?] famous: ... the illusion of choice and you live with it and now you get down to the true, useful 4 variables. Arbitrary vs. useless, every one was put in for a reason.
?: I don't think it is purely an autocratic decision but evolutionary progress. Look at computers with windowing systems, say 1984, there were a whole slew of machines with things you have never heard of with wacky ways of doing things. Most of them died out. Most of the things you do in an Apache config are obsolete, for ways of thinking that are not valid anymore. What will eventually happen, I urge people to reorient thinking, think top-down, task-oriented.
Bartosh [?]: Related but separate problems. From bottom-up how to describe system, has to be some effort on the other side. Can provide foundation for whole system, provide more or less stable framework for services. If Apache is used by 80% of people in room, it will take only once to provide hooks to provide the config file each and every time. ... Had a very similar issue with my users. 90% of what is being done is development so I won't get anyone to fix code to be nice to the infrastructure. The thing that matters is to get published. ... Havoc when someone decides to run 1000 jobs at one time. [talks about example of a major problem to work with developer to take care of this piece using hooks]
Narayan: What you are describing is really differentiation between
what Paul's calling CM and what a lot of people are talking about. What I
want is to say "run apache on this port" and there are a lot of
mechanics that we cannot get away from. Paul gets upset when people talk about
cfengine tidy as the solution. ... I disagree that it is easy once you write a
parser and generator for config files.
Luke ends the CM BoF, transitioning to Puppet BoF.
last modified $Date: 2006/01/13 19:23:57 $