Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1817 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1690. Min Act: -0.0000

Data Index: 13376 (C4 (Web Text))

Max Activating Token Index: 883

Text #1

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1668. Min Act: -0.0000

Data Index: 668332 (C4 (Web Text))

Max Activating Token Index: 816

Text #2

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1700. Min Act: -0.0000

Data Index: 1043432 (C4 (Web Text))

Max Activating Token Index: 780

Text #3

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1439. Min Act: -0.0000

Data Index: 1032819 (C4 (Web Text))

Max Activating Token Index: 233

Text #4

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1464. Min Act: -0.0000

Data Index: 879207 (C4 (Web Text))

Max Activating Token Index: 204

Text #5

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1491. Min Act: -0.0000

Data Index: 1213026 (C4 (Web Text))

Max Activating Token Index: 635

Text #6

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1464. Min Act: -0.0000

Data Index: 658215 (C4 (Web Text))

Max Activating Token Index: 692

Text #7

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1411. Min Act: -0.0000

Data Index: 777835 (C4 (Web Text))

Max Activating Token Index: 454

Text #8

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1406. Min Act: -0.0000

Data Index: 311628 (C4 (Web Text))

Max Activating Token Index: 999

Text #9

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1376. Min Act: -0.0000

Data Index: 323040 (C4 (Web Text))

Max Activating Token Index: 573

Text #10

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1378. Min Act: -0.0000

Data Index: 54905 (C4 (Web Text))

Max Activating Token Index: 771

Text #11

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1353. Min Act: -0.0000

Data Index: 623895 (C4 (Web Text))

Max Activating Token Index: 370

Text #12

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1351. Min Act: -0.0000

Data Index: 1286600 (C4 (Web Text))

Max Activating Token Index: 249

Text #13

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1353. Min Act: -0.0000

Data Index: 727280 (C4 (Web Text))

Max Activating Token Index: 984

Text #14

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1308. Min Act: -0.0000

Data Index: 1104685 (C4 (Web Text))

Max Activating Token Index: 442

Text #15

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1441. Min Act: -0.0000

Data Index: 428117 (C4 (Web Text))

Max Activating Token Index: 819

Text #16

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1346. Min Act: -0.0000

Data Index: 1250818 (C4 (Web Text))

Max Activating Token Index: 494

Text #17

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1277. Min Act: -0.0000

Data Index: 247475 (C4 (Web Text))

Max Activating Token Index: 427

Text #18

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1306. Min Act: -0.0000

Data Index: 562642 (C4 (Web Text))

Max Activating Token Index: 999

Text #19

Max Range: 0.1700. Min Range: -0.1700

Max Act: 0.1287. Min Act: -0.0000

Data Index: 73985 (C4 (Web Text))

Max Activating Token Index: 160