Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3935 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.9136. Min Act: -0.0000

Data Index: 79247 (C4 (Web Text))

Max Activating Token Index: 142

Text #1

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.8570. Min Act: -0.0000

Data Index: 1156663 (C4 (Web Text))

Max Activating Token Index: 510

Text #2

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.8259. Min Act: -0.0000

Data Index: 1293593 (C4 (Web Text))

Max Activating Token Index: 293

Text #3

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.9069. Min Act: -0.0000

Data Index: 935944 (C4 (Web Text))

Max Activating Token Index: 891

Text #4

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.7912. Min Act: -0.0000

Data Index: 1608451 (Python Code)

Max Activating Token Index: 202

Text #5

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.7744. Min Act: -0.0000

Data Index: 716596 (C4 (Web Text))

Max Activating Token Index: 489

Text #6

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.7879. Min Act: -0.0000

Data Index: 1327004 (C4 (Web Text))

Max Activating Token Index: 859

Text #7

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.8015. Min Act: -0.0000

Data Index: 1399951 (Python Code)

Max Activating Token Index: 638

Text #8

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.7617. Min Act: -0.0000

Data Index: 1173300 (C4 (Web Text))

Max Activating Token Index: 384

Text #9

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.7399. Min Act: -0.0000

Data Index: 782866 (C4 (Web Text))

Max Activating Token Index: 321

Text #10

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.6911. Min Act: -0.0000

Data Index: 103224 (C4 (Web Text))

Max Activating Token Index: 472

Text #11

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.7439. Min Act: -0.0000

Data Index: 1576055 (Python Code)

Max Activating Token Index: 389

Text #12

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.7115. Min Act: -0.0000

Data Index: 1567435 (Python Code)

Max Activating Token Index: 336

Text #13

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.7181. Min Act: -0.0000

Data Index: 691093 (C4 (Web Text))

Max Activating Token Index: 388

Text #14

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.7023. Min Act: -0.0000

Data Index: 1523146 (Python Code)

Max Activating Token Index: 925

Text #15

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.7494. Min Act: -0.0000

Data Index: 1402728 (Python Code)

Max Activating Token Index: 229

Text #16

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.6587. Min Act: -0.0000

Data Index: 1407100 (Python Code)

Max Activating Token Index: 46

Text #17

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.6746. Min Act: -0.0000

Data Index: 1467327 (Python Code)

Max Activating Token Index: 698

Text #18

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.6876. Min Act: -0.0000

Data Index: 1585368 (Python Code)

Max Activating Token Index: 370

Text #19

Max Range: 0.9136. Min Range: -0.9136

Max Act: 0.6833. Min Act: -0.0000

Data Index: 1468198 (Python Code)

Max Activating Token Index: 914