Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3036 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0350. Min Act: -0.0003

Data Index: 22793 (C4 (Web Text))

Max Activating Token Index: 722

Text #1

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0344. Min Act: -0.0004

Data Index: 850728 (C4 (Web Text))

Max Activating Token Index: 702

Text #2

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0337. Min Act: -0.0003

Data Index: 851493 (C4 (Web Text))

Max Activating Token Index: 20

Text #3

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0332. Min Act: -0.0003

Data Index: 803231 (C4 (Web Text))

Max Activating Token Index: 417

Text #4

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0336. Min Act: -0.0003

Data Index: 1317058 (C4 (Web Text))

Max Activating Token Index: 543

Text #5

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0319. Min Act: -0.0003

Data Index: 444675 (C4 (Web Text))

Max Activating Token Index: 676

Text #6

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0320. Min Act: -0.0004

Data Index: 148674 (C4 (Web Text))

Max Activating Token Index: 614

Text #7

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0312. Min Act: -0.0003

Data Index: 381654 (C4 (Web Text))

Max Activating Token Index: 349

Text #8

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0311. Min Act: -0.0004

Data Index: 1281832 (C4 (Web Text))

Max Activating Token Index: 496

Text #9

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0299. Min Act: -0.0003

Data Index: 604671 (C4 (Web Text))

Max Activating Token Index: 450

Text #10

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0305. Min Act: -0.0004

Data Index: 733658 (C4 (Web Text))

Max Activating Token Index: 655

Text #11

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0304. Min Act: -0.0003

Data Index: 1035168 (C4 (Web Text))

Max Activating Token Index: 323

Text #12

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0310. Min Act: -0.0003

Data Index: 1054452 (C4 (Web Text))

Max Activating Token Index: 599

Text #13

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0307. Min Act: -0.0004

Data Index: 550804 (C4 (Web Text))

Max Activating Token Index: 628

Text #14

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0300. Min Act: -0.0004

Data Index: 886031 (C4 (Web Text))

Max Activating Token Index: 497

Text #15

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0299. Min Act: -0.0004

Data Index: 1219796 (C4 (Web Text))

Max Activating Token Index: 317

Text #16

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0293. Min Act: -0.0003

Data Index: 838273 (C4 (Web Text))

Max Activating Token Index: 289

Text #17

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0303. Min Act: -0.0003

Data Index: 318346 (C4 (Web Text))

Max Activating Token Index: 456

Text #18

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0296. Min Act: -0.0003

Data Index: 304544 (C4 (Web Text))

Max Activating Token Index: 497

Text #19

Max Range: 0.0350. Min Range: -0.0350

Max Act: 0.0296. Min Act: -0.0003

Data Index: 152715 (C4 (Web Text))

Max Activating Token Index: 712