Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3877 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.5169. Min Range: -3.5169

Max Act: 3.4599. Min Act: -0.0000

Data Index: 1404626 (Python Code)

Max Activating Token Index: 983

Text #1

Max Range: 3.5169. Min Range: -3.5169

Max Act: 3.4240. Min Act: -0.0000

Data Index: 937649 (C4 (Web Text))

Max Activating Token Index: 626

Text #2

Max Range: 3.5169. Min Range: -3.5169

Max Act: 3.5169. Min Act: -0.0000

Data Index: 677988 (C4 (Web Text))

Max Activating Token Index: 354

Text #3

Max Range: 3.5169. Min Range: -3.5169

Max Act: 3.1170. Min Act: -0.0000

Data Index: 255334 (C4 (Web Text))

Max Activating Token Index: 775

Text #4

Max Range: 3.5169. Min Range: -3.5169

Max Act: 3.0239. Min Act: -0.0000

Data Index: 1554557 (Python Code)

Max Activating Token Index: 301

Text #5

Max Range: 3.5169. Min Range: -3.5169

Max Act: 3.0010. Min Act: -0.0000

Data Index: 877828 (C4 (Web Text))

Max Activating Token Index: 571

Text #6

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.9242. Min Act: -0.0000

Data Index: 1415965 (Python Code)

Max Activating Token Index: 273

Text #7

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.7284. Min Act: -0.0000

Data Index: 510586 (C4 (Web Text))

Max Activating Token Index: 571

Text #8

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.6390. Min Act: -0.0000

Data Index: 663218 (C4 (Web Text))

Max Activating Token Index: 68

Text #9

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.5966. Min Act: -0.0000

Data Index: 1057338 (C4 (Web Text))

Max Activating Token Index: 679

Text #10

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.4787. Min Act: -0.0000

Data Index: 941906 (C4 (Web Text))

Max Activating Token Index: 678

Text #11

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.7156. Min Act: -0.0000

Data Index: 1041929 (C4 (Web Text))

Max Activating Token Index: 69

Text #12

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.4685. Min Act: -0.0000

Data Index: 645232 (C4 (Web Text))

Max Activating Token Index: 100

Text #13

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.4450. Min Act: -0.0000

Data Index: 1330433 (C4 (Web Text))

Max Activating Token Index: 342

Text #14

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.4898. Min Act: -0.0000

Data Index: 91535 (C4 (Web Text))

Max Activating Token Index: 130

Text #15

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.3643. Min Act: -0.0000

Data Index: 1296281 (C4 (Web Text))

Max Activating Token Index: 182

Text #16

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.4608. Min Act: -0.0000

Data Index: 502532 (C4 (Web Text))

Max Activating Token Index: 994

Text #17

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.2957. Min Act: -0.0000

Data Index: 741467 (C4 (Web Text))

Max Activating Token Index: 899

Text #18

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.4377. Min Act: -0.0000

Data Index: 862831 (C4 (Web Text))

Max Activating Token Index: 601

Text #19

Max Range: 3.5169. Min Range: -3.5169

Max Act: 2.5239. Min Act: -0.0000

Data Index: 530370 (C4 (Web Text))

Max Activating Token Index: 881