Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3807 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.7541. Min Act: -0.0000

Data Index: 857640 (C4 (Web Text))

Max Activating Token Index: 943

Text #1

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.2760. Min Act: -0.0000

Data Index: 257634 (C4 (Web Text))

Max Activating Token Index: 482

Text #2

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.7764. Min Act: -0.0000

Data Index: 1235825 (C4 (Web Text))

Max Activating Token Index: 706

Text #3

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.8919. Min Act: -0.0000

Data Index: 572899 (C4 (Web Text))

Max Activating Token Index: 207

Text #4

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.8458. Min Act: -0.0000

Data Index: 278119 (C4 (Web Text))

Max Activating Token Index: 557

Text #5

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.0433. Min Act: -0.0000

Data Index: 1070378 (C4 (Web Text))

Max Activating Token Index: 295

Text #6

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.7207. Min Act: -0.0000

Data Index: 1283995 (C4 (Web Text))

Max Activating Token Index: 83

Text #7

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.4856. Min Act: -0.0000

Data Index: 616361 (C4 (Web Text))

Max Activating Token Index: 765

Text #8

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.2661. Min Act: -0.0000

Data Index: 368587 (C4 (Web Text))

Max Activating Token Index: 515

Text #9

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.4468. Min Act: -0.0000

Data Index: 510623 (C4 (Web Text))

Max Activating Token Index: 823

Text #10

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.6139. Min Act: -0.0000

Data Index: 32549 (C4 (Web Text))

Max Activating Token Index: 78

Text #11

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.3889. Min Act: -0.0000

Data Index: 1358937 (C4 (Web Text))

Max Activating Token Index: 460

Text #12

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.5479. Min Act: -0.0000

Data Index: 533770 (C4 (Web Text))

Max Activating Token Index: 144

Text #13

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.5327. Min Act: -0.0000

Data Index: 1105312 (C4 (Web Text))

Max Activating Token Index: 236

Text #14

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.5195. Min Act: -0.0000

Data Index: 71893 (C4 (Web Text))

Max Activating Token Index: 546

Text #15

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.2939. Min Act: -0.0000

Data Index: 411217 (C4 (Web Text))

Max Activating Token Index: 621

Text #16

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.1369. Min Act: -0.0000

Data Index: 708547 (C4 (Web Text))

Max Activating Token Index: 634

Text #17

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.4228. Min Act: -0.0000

Data Index: 587413 (C4 (Web Text))

Max Activating Token Index: 863

Text #18

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.2598. Min Act: -0.0000

Data Index: 463523 (C4 (Web Text))

Max Activating Token Index: 985

Text #19

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.2231. Min Act: -0.0000

Data Index: 246594 (C4 (Web Text))

Max Activating Token Index: 296