Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 826 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 12.6918. Min Range: -12.6918

Max Act: 12.6918. Min Act: -0.0000

Data Index: 1114356 (C4 (Web Text))

Max Activating Token Index: 931

Text #1

Max Range: 12.6918. Min Range: -12.6918

Max Act: 11.8404. Min Act: -0.0000

Data Index: 610863 (C4 (Web Text))

Max Activating Token Index: 242

Text #2

Max Range: 12.6918. Min Range: -12.6918

Max Act: 11.4194. Min Act: -0.0000

Data Index: 666955 (C4 (Web Text))

Max Activating Token Index: 225

Text #3

Max Range: 12.6918. Min Range: -12.6918

Max Act: 11.6475. Min Act: -0.0000

Data Index: 12133 (C4 (Web Text))

Max Activating Token Index: 146

Text #4

Max Range: 12.6918. Min Range: -12.6918

Max Act: 11.2517. Min Act: -0.0000

Data Index: 511239 (C4 (Web Text))

Max Activating Token Index: 621

Text #5

Max Range: 12.6918. Min Range: -12.6918

Max Act: 11.0641. Min Act: -0.0000

Data Index: 876923 (C4 (Web Text))

Max Activating Token Index: 405

Text #6

Max Range: 12.6918. Min Range: -12.6918

Max Act: 11.8292. Min Act: -0.0000

Data Index: 721494 (C4 (Web Text))

Max Activating Token Index: 716

Text #7

Max Range: 12.6918. Min Range: -12.6918

Max Act: 11.2210. Min Act: -0.0000

Data Index: 703320 (C4 (Web Text))

Max Activating Token Index: 87

Text #8

Max Range: 12.6918. Min Range: -12.6918

Max Act: 11.2997. Min Act: -0.0000

Data Index: 1095513 (C4 (Web Text))

Max Activating Token Index: 269

Text #9

Max Range: 12.6918. Min Range: -12.6918

Max Act: 10.8725. Min Act: -0.0000

Data Index: 1175444 (C4 (Web Text))

Max Activating Token Index: 446

Text #10

Max Range: 12.6918. Min Range: -12.6918

Max Act: 10.6736. Min Act: -0.0000

Data Index: 877862 (C4 (Web Text))

Max Activating Token Index: 323

Text #11

Max Range: 12.6918. Min Range: -12.6918

Max Act: 11.0354. Min Act: -0.0000

Data Index: 1035460 (C4 (Web Text))

Max Activating Token Index: 879

Text #12

Max Range: 12.6918. Min Range: -12.6918

Max Act: 10.6283. Min Act: -0.0000

Data Index: 936080 (C4 (Web Text))

Max Activating Token Index: 396

Text #13

Max Range: 12.6918. Min Range: -12.6918

Max Act: 10.4521. Min Act: -0.0000

Data Index: 157588 (C4 (Web Text))

Max Activating Token Index: 649

Text #14

Max Range: 12.6918. Min Range: -12.6918

Max Act: 10.3879. Min Act: -0.0000

Data Index: 1001473 (C4 (Web Text))

Max Activating Token Index: 251

Text #15

Max Range: 12.6918. Min Range: -12.6918

Max Act: 10.1412. Min Act: -0.0000

Data Index: 246410 (C4 (Web Text))

Max Activating Token Index: 115

Text #16

Max Range: 12.6918. Min Range: -12.6918

Max Act: 10.2573. Min Act: -0.0000

Data Index: 752535 (C4 (Web Text))

Max Activating Token Index: 312

Text #17

Max Range: 12.6918. Min Range: -12.6918

Max Act: 10.4906. Min Act: -0.0000

Data Index: 814328 (C4 (Web Text))

Max Activating Token Index: 549

Text #18

Max Range: 12.6918. Min Range: -12.6918

Max Act: 10.2979. Min Act: -0.0000

Data Index: 1170229 (C4 (Web Text))

Max Activating Token Index: 690

Text #19

Max Range: 12.6918. Min Range: -12.6918

Max Act: 9.9780. Min Act: -0.0000

Data Index: 741330 (C4 (Web Text))

Max Activating Token Index: 269