Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3241 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.7639. Min Act: -0.0000

Data Index: 692955 (C4 (Web Text))

Max Activating Token Index: 770

Text #1

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.6699. Min Act: 0.0000

Data Index: 306095 (C4 (Web Text))

Max Activating Token Index: 138

Text #2

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.6140. Min Act: -0.0000

Data Index: 1192558 (C4 (Web Text))

Max Activating Token Index: 921

Text #3

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.6103. Min Act: 0.0000

Data Index: 656817 (C4 (Web Text))

Max Activating Token Index: 489

Text #4

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5769. Min Act: 0.0000

Data Index: 899540 (C4 (Web Text))

Max Activating Token Index: 425

Text #5

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5799. Min Act: -0.0000

Data Index: 546047 (C4 (Web Text))

Max Activating Token Index: 62

Text #6

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5643. Min Act: -0.0000

Data Index: 1243278 (C4 (Web Text))

Max Activating Token Index: 726

Text #7

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5628. Min Act: -0.0000

Data Index: 1130355 (C4 (Web Text))

Max Activating Token Index: 956

Text #8

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5619. Min Act: -0.0000

Data Index: 1010231 (C4 (Web Text))

Max Activating Token Index: 801

Text #9

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5604. Min Act: -0.0000

Data Index: 1343320 (C4 (Web Text))

Max Activating Token Index: 443

Text #10

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5479. Min Act: 0.0000

Data Index: 339389 (C4 (Web Text))

Max Activating Token Index: 343

Text #11

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5269. Min Act: -0.0000

Data Index: 68259 (C4 (Web Text))

Max Activating Token Index: 1019

Text #12

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5279. Min Act: 0.0000

Data Index: 1055492 (C4 (Web Text))

Max Activating Token Index: 963

Text #13

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.4729. Min Act: -0.0000

Data Index: 1327462 (C4 (Web Text))

Max Activating Token Index: 294

Text #14

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5039. Min Act: -0.0000

Data Index: 1268012 (C4 (Web Text))

Max Activating Token Index: 639

Text #15

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5263. Min Act: 0.0000

Data Index: 253056 (C4 (Web Text))

Max Activating Token Index: 153

Text #16

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.4875. Min Act: 0.0000

Data Index: 837857 (C4 (Web Text))

Max Activating Token Index: 17

Text #17

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5016. Min Act: -0.0000

Data Index: 1225766 (C4 (Web Text))

Max Activating Token Index: 682

Text #18

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.5084. Min Act: 0.0000

Data Index: 44502 (C4 (Web Text))

Max Activating Token Index: 226

Text #19

Max Range: 0.7639. Min Range: -0.7639

Max Act: 0.4844. Min Act: 0.0000

Data Index: 677107 (C4 (Web Text))

Max Activating Token Index: 711