Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2547 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.8700. Min Act: -0.0000

Data Index: 1374242 (Python Code)

Max Activating Token Index: 464

Text #1

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.8668. Min Act: -0.0000

Data Index: 1588403 (Python Code)

Max Activating Token Index: 966

Text #2

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.8645. Min Act: -0.0000

Data Index: 1548399 (Python Code)

Max Activating Token Index: 748

Text #3

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.7664. Min Act: -0.0000

Data Index: 1378020 (Python Code)

Max Activating Token Index: 642

Text #4

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.7962. Min Act: -0.0000

Data Index: 1597963 (Python Code)

Max Activating Token Index: 833

Text #5

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.7551. Min Act: 0.0000

Data Index: 1360874 (Python Code)

Max Activating Token Index: 634

Text #6

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.7816. Min Act: -0.0000

Data Index: 1392255 (Python Code)

Max Activating Token Index: 468

Text #7

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.7346. Min Act: 0.0000

Data Index: 1550573 (Python Code)

Max Activating Token Index: 634

Text #8

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.7093. Min Act: -0.0000

Data Index: 1465657 (Python Code)

Max Activating Token Index: 475

Text #9

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.7176. Min Act: -0.0000

Data Index: 1400623 (Python Code)

Max Activating Token Index: 106

Text #10

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.6698. Min Act: -0.0000

Data Index: 1589237 (Python Code)

Max Activating Token Index: 530

Text #11

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.6590. Min Act: -0.0000

Data Index: 1515840 (Python Code)

Max Activating Token Index: 307

Text #12

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.6573. Min Act: -0.0000

Data Index: 1507492 (Python Code)

Max Activating Token Index: 1016

Text #13

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.6405. Min Act: -0.0000

Data Index: 1642716 (Python Code)

Max Activating Token Index: 813

Text #14

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.6410. Min Act: 0.0000

Data Index: 1424085 (Python Code)

Max Activating Token Index: 677

Text #15

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.6427. Min Act: -0.0000

Data Index: 1464391 (Python Code)

Max Activating Token Index: 109

Text #16

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.6199. Min Act: -0.0000

Data Index: 1430472 (Python Code)

Max Activating Token Index: 370

Text #17

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.6333. Min Act: -0.0000

Data Index: 1454109 (Python Code)

Max Activating Token Index: 41

Text #18

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.6290. Min Act: -0.0000

Data Index: 1413181 (Python Code)

Max Activating Token Index: 1010

Text #19

Max Range: 0.8700. Min Range: -0.8700

Max Act: 0.6271. Min Act: -0.0000

Data Index: 1451658 (Python Code)

Max Activating Token Index: 475