[Scamper-dev] scamper notes draft 2

[Ken Keys]

On Wed, May 19, 2004 at 06:32:08PM -0700, Bradley Huffaker wrote:
>     End List Cycle Command - marks the end of a list and causes scamper to
> 	free up resources assigned to it once it has finished the
> 	current cycle.
> 	contains: ListId

End List Cycle Command should also contain CycleId.  (This is not strictly
necessary, as the list manager should never send overlapping cycles, but
serves as a sanity check, and is consistent with every other command
containing ListId,CycleId.)

Also, when the Prober receives an End List Cycle command, it can not free
cycle resources immediately; it has to wait until all addresses from the
cycle are gone from the Active and Hold lists.  I imagine the easiest way
to do this is to have the Cycle state structure have a counter of number
of addresses in active or hold state, and an end_of_cycle_cmd_received
flag.  When an address is done being probed and is removed from the list,
the counter is decremented; if the counter reaches 0 and the
end_of_cycle_cmd_received flag is set, the Cycle state can be freed.

>     Command Queue - queue of commands which were passed by the List Manager
> 	but have yet to be processed

On further discussion, we decided there's no reason for a Command
Queue.  The Prober can just read from the pipe whenever it is ready
for the next command.  Only the Prober knows when it's ready; the
List Manager just generates a sequence of commands, and writes them
to the pipe.  The List Manager will block on write when the pipe
fills because the Prober hasn't read them yet.  Since the communcation
channel is one-way, the full pipe is the only thing making the List
Manager pause.

> When an IP Prob State is removed from the Active Window scamper will first
> check the first command in the Hold List to see if it's Hold Timestamp has
> expired and process the first command in the Hold List.  If the Hold List
> is empty or the first entry's Hold Timestamp has not yet expired it will
> process the next command in the Command Queue.  If the Command Queue is
> empty and the Hold List is not scamper will sleep up to the Hold Timestamp.
> If the Command Queue and the Hold List are empty and there are no futher
> commands from the List Manager scamper will sleep for a short period of
> time before rechecking the Command Queue, say 5 minutes.  It will repeatly
> sleep and check until it is killed or the Command Queue is refilled.

If the Prober uses select() to wait for probe responses, then
whenever the Prober is ready for a new command (i.e., an Active
slot has opened up, it's been long enough since the last probe to
not overstep the max probe rate, and there's nothing eligible in
the Hold List), it can add the command pipe's fd to the select set.
Usually the command pipe is full, so select will immediately indicate
a new command can be read, but if not, select will continue to wait
for probe responses.  No periodic polling is required.

> --------------------------------------------------------------------
>     Data Sorter
> --------------------------------------------------------------------
> 
> The Data Sorter will use the Start Cycle Object to know when to open a 
> new File Handler.  It will use the End Cycle Object to close the file 
> Handler.

Note that two Cycles from the same List may overlap.

-- 
Ken Keys
kkeys at caida.org