Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2642 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2390. Min Act: -0.0000

Data Index: 745322 (C4 (Web Text))

Max Activating Token Index: 677

Text #1

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2297. Min Act: -0.0000

Data Index: 1248906 (C4 (Web Text))

Max Activating Token Index: 847

Text #2

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2259. Min Act: 0.0000

Data Index: 634288 (C4 (Web Text))

Max Activating Token Index: 795

Text #3

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2210. Min Act: 0.0000

Data Index: 1299979 (C4 (Web Text))

Max Activating Token Index: 356

Text #4

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2090. Min Act: -0.0000

Data Index: 114320 (C4 (Web Text))

Max Activating Token Index: 660

Text #5

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2205. Min Act: 0.0000

Data Index: 1000932 (C4 (Web Text))

Max Activating Token Index: 823

Text #6

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2110. Min Act: 0.0000

Data Index: 849772 (C4 (Web Text))

Max Activating Token Index: 409

Text #7

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2012. Min Act: 0.0000

Data Index: 73576 (C4 (Web Text))

Max Activating Token Index: 381

Text #8

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.1933. Min Act: 0.0000

Data Index: 756854 (C4 (Web Text))

Max Activating Token Index: 1023

Text #9

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2027. Min Act: -0.0000

Data Index: 13800 (C4 (Web Text))

Max Activating Token Index: 77

Text #10

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.1969. Min Act: 0.0000

Data Index: 1037135 (C4 (Web Text))

Max Activating Token Index: 536

Text #11

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2002. Min Act: -0.0000

Data Index: 772246 (C4 (Web Text))

Max Activating Token Index: 262

Text #12

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.1982. Min Act: 0.0000

Data Index: 319684 (C4 (Web Text))

Max Activating Token Index: 345

Text #13

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.2027. Min Act: 0.0000

Data Index: 173458 (C4 (Web Text))

Max Activating Token Index: 356

Text #14

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.1976. Min Act: -0.0000

Data Index: 1344510 (C4 (Web Text))

Max Activating Token Index: 611

Text #15

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.1939. Min Act: -0.0000

Data Index: 605280 (C4 (Web Text))

Max Activating Token Index: 246

Text #16

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.1966. Min Act: 0.0000

Data Index: 64409 (C4 (Web Text))

Max Activating Token Index: 546

Text #17

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.1939. Min Act: -0.0000

Data Index: 1200949 (C4 (Web Text))

Max Activating Token Index: 959

Text #18

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.1896. Min Act: -0.0000

Data Index: 1053071 (C4 (Web Text))

Max Activating Token Index: 540

Text #19

Max Range: 0.2390. Min Range: -0.2390

Max Act: 0.1890. Min Act: -0.0000

Data Index: 414205 (C4 (Web Text))

Max Activating Token Index: 30