Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4603 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.7500. Min Act: -0.0000

Data Index: 549391 (C4 (Web Text))

Max Activating Token Index: 49

Text #1

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.7365. Min Act: -0.0000

Data Index: 759121 (C4 (Web Text))

Max Activating Token Index: 152

Text #2

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.7722. Min Act: -0.0000

Data Index: 882907 (C4 (Web Text))

Max Activating Token Index: 452

Text #3

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.7046. Min Act: -0.0000

Data Index: 1262912 (C4 (Web Text))

Max Activating Token Index: 219

Text #4

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.7178. Min Act: -0.0000

Data Index: 418123 (C4 (Web Text))

Max Activating Token Index: 666

Text #5

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.7792. Min Act: -0.0000

Data Index: 223835 (C4 (Web Text))

Max Activating Token Index: 118

Text #6

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.7585. Min Act: -0.0000

Data Index: 1247543 (C4 (Web Text))

Max Activating Token Index: 542

Text #7

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6554. Min Act: -0.0000

Data Index: 938204 (C4 (Web Text))

Max Activating Token Index: 553

Text #8

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6984. Min Act: -0.0000

Data Index: 527412 (C4 (Web Text))

Max Activating Token Index: 1003

Text #9

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.7006. Min Act: -0.0000

Data Index: 1245494 (C4 (Web Text))

Max Activating Token Index: 287

Text #10

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6735. Min Act: -0.0000

Data Index: 121999 (C4 (Web Text))

Max Activating Token Index: 929

Text #11

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.7013. Min Act: -0.0000

Data Index: 690419 (C4 (Web Text))

Max Activating Token Index: 802

Text #12

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6003. Min Act: -0.0000

Data Index: 138486 (C4 (Web Text))

Max Activating Token Index: 396

Text #13

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6489. Min Act: -0.0000

Data Index: 203781 (C4 (Web Text))

Max Activating Token Index: 803

Text #14

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6315. Min Act: -0.0000

Data Index: 1230236 (C4 (Web Text))

Max Activating Token Index: 191

Text #15

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6152. Min Act: -0.0000

Data Index: 987252 (C4 (Web Text))

Max Activating Token Index: 600

Text #16

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6531. Min Act: -0.0000

Data Index: 1042502 (C4 (Web Text))

Max Activating Token Index: 466

Text #17

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6537. Min Act: -0.0000

Data Index: 744289 (C4 (Web Text))

Max Activating Token Index: 931

Text #18

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6276. Min Act: -0.0000

Data Index: 326753 (C4 (Web Text))

Max Activating Token Index: 895

Text #19

Max Range: 0.7792. Min Range: -0.7792

Max Act: 0.6162. Min Act: -0.0000

Data Index: 1231862 (C4 (Web Text))

Max Activating Token Index: 412